When reversing a vehicle, both the cognitive load on a driver and the collision risk may vary for a number of reasons. For example, when backing a vehicle and/or trailer into a parking space, the driver may need to monitor front corners of the vehicle in addition to the sides and rear of the vehicle to assure there is enough clearance between the vehicle and surrounding objects. In addition, the surrounding objects may also be moving, such as when a bicycle crosses the projected path of the vehicle. The cognitive load of the driver generally increases with vehicle speed due to the heightened situational awareness needed by the driver in order to operate the vehicle at that speed. While there may be perimeter sensing and collision mitigation features in place to warn the driver and stop the vehicle, it may be desirable to limit the speed of the vehicle to help manage the cognitive load of the driver and provide more comfortable collision mitigation when there are distracting conditions that may be competing for the driver's attention. Furthermore, managing the vehicle speed based on sensor activity can help limit the risk of a collision even with an automated driving system in place.